Non-equilibrium thermodynamics of stochastic systems with odd and even variables 



O 
0-) 



-a- 
o 

o 

ON' 

o 



X: 



Richard E. Spinney and Ian J. Ford 
Department of Physics and Astronomy, UCL, Gower Street, London WCIE 6BT, UK and 
London Centre for Nanotechnology, 17-19 Gordon Street London WCIH OAH, UK 

(Dated: December 20, 2011) 

The total entropy production of stochastic systems can be divided into three quantities. The first 
corresponds to the excess heat, whilst the second two comprise the house- keeping heat. We denote 
these two components the transient and generalised house-keeping heat and we obtain an integral 
fluctuation theorem for the latter, valid for all Markovian stochastic dynamics. A previously reported 
formalism is obtained when the stationary probability distribution is symmetric for all variables that 
are odd under time reversal which restricts consideration of directional variables such as velocity. 
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For over 100 years the statement of the second law of 
thermodynamics stood simply as the Clausius inequality. 
However in recent years advances in technology have en- 
couraged the thermodynamic consideration of small sys- 
tems which has led to the generalisation of the concept of 
entropy production: it may be associated with individ- 
ual dynamical realisations revealing a wealth of relations 
valid out of equilibrium. Such extensions had their ori- 
gins in the dissipation function of Evans et al. for ther- 
mostatted systems that led to the Fluctuation Theorem 
[ll-Q with similar, but asymptotic relations for chaotic 
systems Q which were extended to Langevin dynam- 
ics followed by general Markovian stochastic systems 
0. Crooks and Jarzynski [§-[13 then derived work rela- 
tions for a variety of dynamics which held for finite times. 
These were followed by similar generalised relations for 
the entropy production associated with transitions be- 
tween stationary states [lli, the total entropy produc- 
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12j and the heat dissipation required to maintain 
a stationary state [13] • More recently the relationshir 
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between the latter quantities has been explored 
resulting in a formalism involving a division of the total 
entropy change into two distinct terms, the adiabatic and 
non-adiabatic entropy productions [l8l - l20l | , each of which 
obeys appropriate fluctuation relations and which map 
onto the house-keeping and excess heats, respectively, of 
Oono and Paniconi [2l|. We seek to take such a formal- 
ism and generalise its scope by the explicit inclusion of 
both even (e.g. spatial) and odd (e.g. momentum) vari- 
ables that transform differently under time reversal. In 
doing so we define a new quantity which obeys an integral 
fiuctuation theorem for all time. 



Specifically, we consider the dynamics of a general set 
of variables x = (x^, x^, . . . x") that behave differently 
under time reversal such that ex = (e^x^, e^a;^, . . . e"a;") 
where = ±1 for even and odd variables respectively. 
Odd variables arise in the discussion of directional quan- 
tities and consequently such a consideration is essential 
when discussing velocities, from the most simple lattice 
Boltzmann model to considerations of full phase space. 
The entropy production of a path of duration r depends 



on two probabilities. The first is the path probability, 
defined as the probability of the forward trajec- 
tory, X = x{t) for < t < T, with a distribution of 
starting configurations, P^(a;(0),0), that acts as an ini- 
tial condition for the general master equation (relevant 
examples arise, for example, in the context of full phase 
space [H, 23| and in lattice Boltzmann models): 



dP^{x,t) 
dt 



Y^T{x\^,\^{t))P''{^,t) 



(1) 



where T(a;|a/, A^(t)) is a matrix of transition rates be- 
tween configurations a/ and a;, defining the normal dy- 
namics, parametcriscd by the forward protocol at time 
t. We use notation T(a;|a;) = — X^x'^^a; ^(^1^) which de- 
scribes the mean escape rate. The path probability of 
some sequence of N transitions to configurations xi from 
Xi-i at times ti, such that to = and ^at+i = r, can 
then be computed as a function of transition rates and 
exponential waiting times 



N 

X WT{x,\x,.i,\^{t,))dUe 



(2) 



We compare this probability to that of another trajectory 
X* , protocol A*, initial condition P*(a;*(0),0) and chosen 
dynamics, denoted P*, and write 



A[x] [P^[x]/P*[x]] 



(3) 



Such a quantity may obey an integral fiuctuation theorem 
(IFT) which may be derived by explicit summation over 
all possible paths, x, for which 7^ as follows 



(exp [-A[x]]f = J2 ^^[^] [-^[^]] 

X 

= J2p*[x*] = 1. 



P*[x] 
(4) 



We assume a one to one mapping between x and x* (a 
condition equivalent to a Jacobian of unity in the trans- 
formation) so that we may consider the summation over 
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5* to be equivalent to that over x. We also require that 
P*[5*] — for all P^[5] = such that the final summa- 
tion contains all possible paths 5* , meaning the required 
normalisation of P*[5*] then yields the result of unity. 
A key result is the implication > by Jensen's 

inequality. 

A common choice for P*, and that used to construct 
the total entropy production, is that of the normal dy- 
namics under the reversed protocol, denoted P* = P^. 
Given the specification of the normal dynamics we point 
out that all further specifications, including the choice 
of protocol, can be systematically derived from the ap- 
propriate path transformation x* which we must choose 
carefully in conjunction with the dynamics so as to obey 
the above conditions. At this point we must be clear 
that given a transition a; — > a/ under the normal dy- 
namics, the transition a/ — > a; is not, in general, possi- 
ble under those same dynamics. Explicitly, we can con- 
struct models such that T{ai\x) ^ whilst T{x\x!) = 
(as an intuitive example: Hamiltonian dynamics can- 
not produce a negative positional step whilst the ve- 
locity is positive). The correct path, x* , to consider is 
the time reversed trajectory proper which includes a re- 
versal of sign for all odd variables. This is the choice 
x*{t) = x''{t) = ex{T — t) and it satisfies the condition 
P*[x] = P^lx'] = for all P^[x] = required for an 
IFT. The reversed protocol A* = A" may be similarly ob- 
tained from the forward protocol, which may be treated 
as an even dynamical variable, meaning it transforms to 
yield A*(t) = eX^{T-t) = X^{T~t) = X^{t). And fi- 
nally we require the choice of initial condition for the 
reverse path. This may be informed physically: we seek 
to characterise the irreversibility of the forward path and 
so initiate the reverse behaviour by time reversing the 
coordinates, x{t), and distribution, P^(a;(T),T), at the 
end of the forward process and evolve forward in time 
from there. The distribution can also be found by apply- 
ing the transformation rules used to obtain the trajec- 
tory x' from X such that P*(a;*(0),0) = P^(a;t(0),0) = 
iP^{ex{T),T) = P^ {eex{T),T) — P^(a;(T),r) where i 
denotes the time reversal operation on the distribution. 
In this instance the path probability is therefore 

N ^ 

X nr(4l4_i,A«(t,))rfi,e^*r'''*'^(--l---^"(*'». (5) 

1=1 



We have xj = exj^^i so we may rearrange to give 

P [x'\ = P {XN,T)e-"'^ ^ ' ^ " 



(6) 
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T{exi^i\£Xi, X^ {tN^i^i))dti 



X^ {ti) — A^(tjv_i+i) such that 
Pi^[5^] = pF(a;jv, T)e- 'i*'^(-o|e.o,A-(t')) (7) 

J|e-^'Vi'^*'"'(^^'l^^-''(*'»r(£a.,_i|£a=,:,A^(t,))dt.. 
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A comparison of P^[a;] and P'^[a;^] characterises the ir- 
reversibility of the forward path and defines the total 
entropy production (using units feg = 1) 



A^tot =lnP''[5] -lnP" [5^] 
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P^{xn,t) ■ f^''\f^;+' dt'T{ex,\ex,,\''{t')) 



r(a;,|a^_i,A^ {U)) 

T{£X,^l\£X„X^{ti)) 



which by its definition and Eq. (jlj obeys 12 1 
(exp[-A5tot]>^ = l. 



(8) 



(9) 



We find that this form of ASjjjt is more complicated 
than previous descriptions 3 2J1 unless ex = x. Note 
that if detailed balance holds, such that P'^'^(a;)T(a/|a;) = 
P'''i(£a/)r(£a;|£a/), we expect P°i, the equilibrium state 
for a given A^(<), to satisfy P'"^(a;) = P'''^(£a;) due to 
time-reversal invariance, along with r(a;|a;) = T(£a;|£a;). 
For a system in equilibrium, we therefore conclude that 
AS'tot = for all paths. 

Next we consider alternative specifications of P*. We 
consider the adjoint dynamics which lead to the same 
stationary state. P*'*(a;, A^(i)), as the normal dynamics, 
but generate flux of the opposite sign in that stationary 
state. It can be shown [3, [3, [i^l that this requires an 
adjoint transition rate matrix T^^ described by 



r"^(a;|a;',A^ {t))^T{x!\x, X' {t}) 



P"*(a;, A^(t)) 
P^t(a^,AF(t))' 



(10) 



However, in the same way that the normal dynamics may 
not, in general, permit transitions xf — > a; or £a; — > ex! , 
similarly the adjoint dynamics may not, in general, per- 
mit transitions a;— > a/ or ex! — £a;. Thus we must 
consider the representation of the adjoint dynamics as 
either Eq. (HU]) or 



T-(£.'|£.,A-(t))=T(£.|£.',A-(t))^Ml^ 



(11) 



We then perform a change of variable t' — > r — and use 



depending on the specific transition being considered. 
Explicitly, when choosing P*[5*], we should not consider 
P'"^[5] or P'^'^[S^] since these might violate the required 
condition P*[x*] = for all P^[2] = 0, required for an 
IFT. 

Under the adjoint dynamics, however, an appropriate 
transformation of x is x*{t) = a^ {t) = x{T—t). Applying 
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the transformation rules used to obtain x yields the 
reverse protocol as before A*(t) = A^(r — = A^(t) and 
the initial distribution P*(a;*(0),0) = F^^^^i^ (aJ^(O), 0) = 
{x(t)^t). The path probability is then 

4=1 

= r)e- rft'T-(xo|™,A-(0) (12) 



We then construct a quantity of the form given in Eq. , 
utihse Eq. dTUl) and the property T'^^{x\'£) =T{x\x), valid 
by means of balance, to obtain 



A5i = lnP^ [5] -InP'^^-^p^] 



which through its definition and Eq. (j4]) obeys 
(exp [-AS,]f = 1 



(13) 



(14) 



which exists in the literature as the Hatano-Sasa rela- 
tion 11 ll or IFT for the non-adiabatic entropy produc- 
tion |18l420l |. Let us now consider, once again under 
the adjoint dynamics, the path transformation choice 
x*(t) = x^{t) = ex{t). Applying the transformation 
rules we obtain the protocol X*{t) = eA^(t) = A^(t) 
and initial distribution P*(a;*(0),0) = P^'^^'^(a;'^(0), 0) = 
£P^(£a;(0),0) = P'^(a;(0),0). The path probability for 
this case is therefore 



N 



X JlT'^''{xJ\xt,,\^{U))dUe^^: .Zt'T-(x7|x7.A-(t')) 

i=l 

pP(a^,0)e/4^^*'^°'(-«l-^-^"(*')) (15) 
X n7^'''(^a;,|£a^_i, AF(t.))dt.e^'^'^' 



By Eq. Q this then allows us to define 
A^a = lnP^[S] - \nP'"^'^[x^] 



^ p/,;+^<it'T(x.|x„A^(t')) 

=yinA^ 

A P-\ex,^i,\''{U)) r(a;,|x,_i,AF(t,)) 
^ P^*(£a;„AF(tO) T(ea^-i|£a^,AF(tO) ^ ^ 

which similarly must obey 

(exphA52])P = l. (17) 



Unlike AS'i, the quantity AS'2 is new in the literature. 
We must immediately recognise that AS'tot 7^ AS'i + A52 
differing by a quantity 



AS3 



N 

E 

i=l 



, P-Ha;.-i,AF(tO)P-Hea;„AP(t,)) 
''p^t(a^,AF(tO)i^^'(ea;.-i,AF(tO) ^ ^ 



such that AS'tot = ASi + AS2 + AS3. If ea; x 
then AS3 = and AS2 reduces to the adiabatic en- 
tropy production appearing in [l8l - [20| . More importantly 
we must recognise that AStot — ASi = AS2 -I- AS3 = 
lj^pad,R[jR] _inpR[2t] or AStot -AS2 = ASi + AS3 = 
■|^j-i^pad,F|-jgT-| _i]2pR[jtj cai^not be written in the form re- 
quired for Eq. (jH) and so do not obey an IFT and do not 
necessarily have any bounds on the sign of their mean. 
We proceed by following the formalism of Seifert [l2. 25| 
and write 



ASt, 



In 



pF(a;(0),0) AQ 

P^{x{t),t) Penv 



ASgys 



AQ 
T 

J- nnv 



(19) 



where Tonv is the temperature of the environment, and 
that of Oono and Paniconi, such that total heat trans- 
fer to the environment, AQ, is the sum of the excess 
heat and house- keeping heat AQ = AQcx + AQhk [21 1 . 
The house-keeping heat is associated with the entropy 
production in stationary states and arises from a non- 
equilibrium constraint that breaks detailed balance. The 
sum AS2 + AS3 is manifestly the entropy production in 
the stationary state and since we are considering Markov 
systems, both AS2 and AS3 are only non-zero when de- 
tailed balance is broken. Hence it is sensible to associate 
AS2 + AS3 with the house-keeping heat such that 



AQhk = (AS2+AS3)re, 



(20) 



ASi is zero for all trajectories in the stationary state 
consolidating the definition of the excess heat as the heat 
transfer associated with an entropy flow that exactly can- 
cels the change in system entropy in the stationary state 
such that 



AQe 



(ASi-ASsys)Te, 



(21) 



However, the prevailing definition of the house-keeping 
heat does not make clear its properties when the system 
is not in a stationary state. A reported formalism sug- 
gests that it is associated with the adiabatic entropy pro- 
duction which serves as a g eneral measure of the break- 
age of detailed balance [18l420j . When considering cases 
where ex= x, this is a consistent approach and the mean 
house-keeping heat obeys strict positivity requirements 
suggesting the entropy additively increases due to non- 
equilibrium constraints and a lack of detailed balance on 
top of that arising from relaxation. However, with the 
inclusion of odd variables this simple picture no longer 
holds, with an ambiguity illustrated by the fact that any 
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of AS'2, AS'3 or AS'2+AS'3 could be argued to be a mea- 
sure of the departure from detailed balance. In the light 
of Eq. ([T7)) we propose that it is sensible to divide the 
house-keeping heat into two quantities which map onto 
AS'2 and AS3. It is important to observe that, on aver- 
age, the rate of change of AS3 vanishes in the stationary 
state by means of balance: the path integral over an 
increment in AS3 explicitly vanishes. Consequently we 
define the 'transient house-keeping heat' and the 'gener- 
alised house-keeping heat' 



r(X,. + IZ.+) = -B-C A5, =0 AS, = (A-B)Ar 



Ag 



hk,T 



= AS3T, 



3-*^ cnv 



AQ 



hk.G 



2 J cnv 



(22) 



T(X- 


-\X,+) = B 


AS,= 


21n(A/B) 


AS, = 


-ln(A/B) 




r(A',,,+ix,,+) 

AS, =0 AS, ^ 




T(X-IX,^-) = C 
AS, =0 AS, = 



T{X-\X-) = -A-C AS,=0 AS,=-{A-B)At 



such that AQhk = AQhk.T + AQhk.G- Since 
(dASs/dr)^'^^ = 0, the generalised house-keeping heat, 
when averaged, has the mean properties previously at- 
tributed to the house-keeping heat: it describes the heat 
flow required to maintain a non-equilibrium stationary 
state and is rigorously non-negative. Our central result 
therefore is 



(exp[-AQhk,G/Tcnv])^ = 1 



(23) 



so (AQhk.G)^ > for all times, protocols and initial con- 
ditions. As a corollary we also state that in general 



(exp[-AQhk/Te„v])^^l 



(24) 



providing no bounds on (AQhk)^ except in the station- 
ary state when ASi = and AQhk/Tenv = AStot or 
generally when P^'^iex, (t)) = P^'^x, \^ {t)) . As such 
the view that the mean rate of entropy production is the 
sum of two specific non-negative contributions as in [1- 
[20l |. is incomplete. The contribution associated with a 
non-equilibrium constraint requires further unravelling, 
particularly when out of stationarity. 

To explore the nature of the house-keeping heat we 
consider its behaviour in the approach to the station- 
ary state of a simple model of particle dynamics on a 
ring. The phase space consists of L identical spatial po- 
sitions Xi , X2 ■ ■ ■ Xl and two velocities labelled + and 
— as shown in Fig. [1] with the time reversal properties 
eXi± ~ Xi^ necessitated by the one-way nature of many 
of the transitions. The stationary state probabilities that 
arise from these dynamics are P^^{Xi+) = A/ {L{A + B)) 
and P"*(Xi-) = B/{L{A + B)). Any difference between 
the velocity reversal rates A and B gives rise to a non- 
equilibrium stationary state by providing a stationary 
particle current, which ioT A > B runs from left to right. 
Such dynamics amount to a very simple lattice Boltz- 
mann model. Contributions AS'2 and AS3 associated 
with particle behaviour consisting of instantaneous tran- 
sitions and waiting periods are indicated. We consider 
particle behaviour over a small time interval dt, and com- 
pute the mean entropy production rates to leading order 
in dt. Examining the path probability in Eq. ^ we need 
only consider iV = or A'^ = 1 transitions. Identifying 



FIG. 1. Allowed moves between positions Xi and ± velocity 
states are shown by arrows, with associated rates T. Peri- 
odic boundaries allow jumps from Xl+ to Xi+ and Xi— to 
Xl — - A given path contributes to the transient and gen- 
eralised house-keeping heats, TonvASa and TonvA^, respec- 
tively, due to transitions between, and residence times At at, 
each phase space point, as indicated. These correspond to 
individual terms in the summations in Eqs. (|16|) and (|18|l . 



leading order terms in the products of P, T, exponenti- 
ated waiting times and AS3 that make up the average of 
the form given in Eq. (|4]) yields 



djASs)' 
dt 



2P(X,+)S In ^ + 2PiX,-)A \n^. 

1=1 



A' 



(25) 

For non-stationary P its sign is unbounded: for example 
if all the probability were uniformly distributed initially 
amongst the -I- velocity states it would equal 2B\n{A/ B), 
whilst if it were distributed over the — states it would 
be —2A\n{A/ B) instead. Such non-zero contributions 
to AS3 require an asymmetric stationary state in odd 
variables which thus explains their absence when the sta- 
tionary velocity distribution is assumed to be symmetric, 
such as in overd amp ed Langevin descriptions (see [isj 
and examples in [20[). However, in the stationary state 
with P = P^^, d{AS3)^ / dt is demonstrably equal to zero 
as claimed. By similar means 



djAS^)' 
dt 



1=1 

+ P{x-) 



A-B-B\n — 
B 



B-A~A\n^ 
A 



(26) 



which is positive for all positive A and B and reduces 
to d(AS2)^'"Vdt = {A - Bf/{A + B) in the station- 
ary state. We note that the sum of Eqs. ((25)) and ([26)) 
has no bound on its sign and relates to the inequal- 
ity in Eq. ([24]). Further, d(exp [-AS2])''/di = and 
(exp [—AS2{t = 0)])^ = 1 which explicitly demonstrates 
the expected IFT for any normalised P{Xi±). Finally, 
we note that for A = B, all contributions vanish in de- 
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tail as this corresponds to equilibrium where there is no 
entropy production. 



We have extended the formalism found in [11|, |13|, [18 - 
[20I and split the total entropy production into two rig- 
orously positive contributions and a third contribution 
which has no bounds on its sign. We have argued that 
this final quantity is, in the mean, a transient contri- 
bution to the house-keeping heat and it is the mean 
generalised house-keeping heat that is rigorously posi- 
tive for all times. It is not straightforward to consoli- 
date this with the two causes of time reversal asymmetry 
namely relaxation to the stationary state and imposed 
non-equilibrium constraints: AS^ exists only in the pres- 
ence the latter, but is, in the mean, its own measure of re- 
laxation to the stationary state. It could be argued that 
the non-adiabatic entropy production and Hatano-Sasa 
relation do not fully capture the entropy production due 
to transitions between stationary states, but associating 
AS3 with one or other form of entropy production is not 
entirely satisfactory as it occurs when the line between 
them is blurred. Nevertheless, either interpretation elu- 
cidates a new layer of complexity in the theory of entropy 
production in stochastic systems. Further exploration in 
the context of continuous stochastic processes is to be re- 
ported elsewhere [2^ . The authors acknowledge financial 
support from EPSRC. 
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